Subset Selection with Shrinkage: Sparse Linear Modeling When the SNR Is Low

Learning Compact High-Dimensional Models in Noisy Environments Building compact, interpretable statistical models where the output depends upon a small number of input features is well-known problem modern analytics applications. A fundamental tool used this context prominent best subset selection (BSS) procedure, which seeks to obtain linear fit data subject constraint on nonzero features. Whereas BSS procedure works exceptionally well some regimes, it performs pretty poorly out-of-sample predictive performance when underlying are noisy, quite common practice. In paper, we explore relatively less-understood overfitting behavior low-signal noisy environments and propose alternatives that appear mitigate such shortcomings. We study theoretical properties our proposed regularized show promising computational results various sets, using tools from integer programming first-order methods.